Low power and low data-rate imager

ABSTRACT

An imaging system for low-power and low data-rate applications is provided. The imaging system comprises a pixel array having a plurality of photosensitive elements (pixels) divided into a plurality of groups of photosensitive elements (super pixels). An image processor is operably connected to the pixel array and configured to selectively operate each group of photosensitive elements in either (i) a high resolution mode in which the pixel array outputs readout voltages corresponding to all of the photosensitive elements in the respective group of photosensitive elements or (ii) a low resolution mode in which the pixel array outputs readout voltages corresponding to only a subset of the photosensitive elements in the respective group of photosensitive elements. Groups of photosensitive elements corresponding to detected motion in each image frame are operated in the high resolution mode, while the remaining groups of photosensitive elements are operated in the low resolution mode.

This application claims the benefit of priority of U.S. provisionalapplication Ser. No. 62/543,646, filed on Aug. 10, 2017, and of U.S.provisional application Ser. No. 62/547,297, filed on Aug. 18, 2017, thedisclosures of which are herein incorporated by reference in theirentirety.

FIELD

The device and method disclosed in this document relates to an imagingsystem and, more particularly, to an imaging system which achieves lowpower and low data rate without losing important information.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to the prior art by inclusion in this section.

Many image processing algorithms rely on background subtraction toidentify moving objects. In such algorithms, a background model is firstgenerated, which is then used to identify objects. Since the backgroundmodel changes due to the change of objects such as change in lightingcondition, movement of objects such as chairs and so forth, thebackground model has to be constantly updated. Many image processingalgorithms operate on small blocks of the pixel array as opposed to onevery single pixel. That is, instead of labeling each pixel as an objector background, some algorithms focus on a small region to identify ifthere is an object in that region. Such small regions are called “superpixels” and the design of the image sensor pixel array is also based onsuch super pixels. Generally imagers with on-chip or in-pixel processingof data exist in the market today. However, none of these imagers havingon-chip or in-pixel processing enable reading out only relevant datafrom the pixel array in an attempt to reduce the data rate and also thepower consumption.

SUMMARY

An imaging system is disclosed. The imaging system comprises a pixelarray having a plurality of photosensitive elements arranged in a gridformation and being divided into a plurality of groups of photosensitiveelements, the photosensitive elements in each group of photosensitiveelements being adjacent to one another in the grid formation, the pixelarray being configured to output readout voltages proportional to lightintensities at individual photosensitive elements in the plurality ofphotosensitive elements; an analog-to-digital converter operablyconnected to the pixel array to receive the readout voltages and convertthe readout voltages into digitized light intensity values; and at leastone processor operably connected to the pixel array and theanalog-to-digital converter, the at least one processor being configuredto selectively operate each group of photosensitive elements in theplurality of groups of photosensitive elements in one of (i) a highresolution mode in which the pixel array outputs readout voltagescorresponding to all of the photosensitive elements in the respectivegroup of photosensitive elements and (ii) a low resolution mode in whichthe pixel array outputs readout voltages corresponding to only a subsetof the photosensitive elements in the respective group of photosensitiveelements, wherein group of photosensitive elements in the plurality ofgroups of photosensitive elements is operable by the at least oneprocessor in both the high resolution mode and the low resolution modeat different times.

In one embodiment, the at least one processor is configured to: receivean image frame, from the analog-to-digital converter, comprisingdigitized light intensity values corresponding to at least some of thephotosensitive elements in each group of photosensitive elements in theplurality of groups of photosensitive elements; and selectively operateeach group of photosensitive elements in the plurality of groups ofphotosensitive elements in the high resolution mode and the lowresolution mode at different times depending on the digitized lightintensity values of the image frame.

In one embodiment, the at least one processor is configured to: operatea first group of photosensitive elements in the plurality of groups ofphotosensitive elements in the low resolution mode depending on to thedigitized light intensity values in the image frame corresponding to thephotosensitive elements in the first group of photosensitive elements;and operate a second group of photosensitive elements in the pluralityof groups of photosensitive elements in the high resolution modedepending on to the digitized light intensity values in the image framecorresponding to the photosensitive elements in the second group ofphotosensitive elements.

In one embodiment, the at least one processor is configured to, for eachgroup of photosensitive elements in the plurality of groups ofphotosensitive elements: detect whether any moving objects are presentin a respective portion of the image frame corresponding to therespective group of photosensitive elements based on the digitized lightintensity values in the image frame corresponding to the photosensitiveelements in the respective group of photosensitive elements; operate therespective group of photosensitive elements in the low resolution modein response to no moving object being detected in the respective portionof the image frame corresponding to the respective group ofphotosensitive elements; and operate the respective group ofphotosensitive elements in the high resolution mode in response to amoving object being detected in the respective portion of the imageframe corresponding to the respective group of photosensitive elements.

In one embodiment, the at least one processor is configured to, for eachgroup of photosensitive elements in the plurality of groups ofphotosensitive elements, compare the respective portion of the imageframe corresponding to the respective group of photosensitive elementswith a corresponding portion of a reference background image frame todetect whether any moving objects are present in the respective portionof the image frame.

In one embodiment, the at least one processor is configured to generatean intermediate image frame comprising light intensity difference valuesby subtracting the reference background image frame from the receivedimage frame.

In one embodiment, the at least one processor is configured to, for eachgroup of photosensitive elements in the plurality of groups ofphotosensitive elements: operate the respective group of photosensitiveelements in the low resolution mode in response to the absolute value ofeach light intensity difference value of a respective portion of theintermediate image frame that corresponds to the respective group ofphotosensitive elements being less than a predetermined thresholddifference value; and operate the respective group of photosensitiveelements in the high resolution mode in response to the absolute valueof any light intensity difference value of a respective portion of theintermediate image frame that corresponds to the respective group ofphotosensitive elements being greater than a predetermined thresholddifference value

In one embodiment, the at least one processor is configured to: detect achange of lighting conditions in the image frame based on the digitizedlight intensity values of the image frame; and operate the pixel arraywith an adjusted exposure time that depends on the change in thelighting conditions.

In one embodiment, the at least one processor is configured to identifyat least one portion of the reference background image frame to beupdated based on the intermediate image frame, the at least one portioncorresponding to at least one group of photosensitive elements in theplurality of groups of photosensitive elements; and update the at leastone portion of the reference background image frame based on at leastone corresponding portion of the image frame.

In one embodiment, each group of photosensitive elements in theplurality of groups of photosensitive elements comprises a common numberof photosensitive elements arranged in a common formation within thegrid formation and, for each group of photosensitive elements in theplurality of groups of photosensitive elements that is operated in thelow resolution mode, the subset of the photosensitive elements for whichthe pixel array outputs readout voltages corresponds to a definedpattern of photosensitive elements within the common formation.

In one embodiment, the defined pattern of photosensitive elements withinthe common formation is adjustable by the at least one processor.

In one embodiment, the at least one processor is configured to: for eachgroup of photosensitive elements in the plurality of groups ofphotosensitive elements that is operated in the low resolution mode,control the analog-to-digital converter to convert the readout voltagescorresponding to the respective the subset of the photosensitiveelements in the respective group of photosensitive elements with a firstbit depth that is less than a maximum bit depth of the analog-to-digitalconverter; and for each group of photosensitive elements in theplurality of groups of photosensitive elements that is operated in thehigh resolution mode, control the analog-to-digital converter to convertthe readout voltages corresponding to all of the photosensitive elementsin the respective group of photosensitive elements with the maximum bitdepth of the analog-to-digital converter.

In one embodiment, the first bit depth is adjustable by the at least oneprocessor.

In one embodiment, the at least one processor is configured to: for eachgroup of photosensitive elements in the plurality of groups ofphotosensitive elements that is operated in the low resolution mode,control the pixel array and the analog-to-digital converter to updateand convert the readout voltages corresponding to the subset of thephotosensitive elements in the respective group of photosensitiveelements with a first update rate that is less than a maximum updaterate; and for each group of photosensitive elements in the plurality ofgroups of photosensitive elements that is operated in the highresolution mode, control the pixel array and the analog-to-digitalconverter to update and convert the readout voltages corresponding toall of the photosensitive elements in the respective group ofphotosensitive elements with the maximum update rate.

In one embodiment, the first update rate is adjustable by the at leastone processor.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and other features of the imaging system areexplained in the following description, taken in connection with theaccompanying drawings.

FIG. 1 shows an exemplary embodiment of an imaging system.

FIG. 2 shows exemplary components of the imaging system of FIG. 1.

FIG. 3 shows an exemplary detailed view of the arrangement of theindividual sensor elements of the pixel array.

FIG. 4A shows operation of a super pixel of the pixel array in a lowresolution mode.

FIG. 4B shows operation of a super pixel of the pixel array in a highresolution mode.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thedisclosure, reference will now be made to the embodiments illustrated inthe drawings and described in the following written specification. It isunderstood that no limitation to the scope of the disclosure is therebyintended. It is further understood that the present disclosure includesany alterations and modifications to the illustrated embodiments andincludes further applications of the principles of the disclosure aswould normally occur to one skilled in the art which this disclosurepertains.

With reference to FIGS. 1-2, exemplary embodiments of an imaging system10 are shown. As shown in FIG. 1, the imaging system 100 at leastincludes a pixel array 110 (which may also be referred to herein as a“sensor array”). The pixel array 110 is configured to capture aplurality of image frames of a scene 20 over time. It will beappreciated that, at a given position and orientation of the pixel array120, the pixel array 110 has a restricted field of view 30 such that animage captured by the pixel array 110 includes only a portion of thescene 20. The scene 20 comprises a real-world environment and, as such,may contain a variety of objects or structures therein. Generally, thescene 20 includes objects or structures 40 (e.g., buildings, trees,walls, furniture, etc.) that are static or mostly static, which make upthe background environment of the scene 20. In contrast, the scene mayalso include moving objects 50 (e.g., people, animals, cars, etc.),which may move into or out of the field of view 30 of the pixel array110 over time. For example, a moving object 50 may move along a path 60through the scene 20. As illustrated, the moving object 50 has movedinto the field of view 30 of the pixel array 110. It will beappreciated, however, that the distinction between the static objects 40and the moving objects 50 is dynamic and merely for illustrativepurposes. In some cases, the moving object 50 may move stay in motionwhile it is in the field of view 30. In other cases, the moving object50 may stop moving for some time and become static (such as a person whomoves into the field of view 30 and sits down). Similarly, a staticbackground object 40 may move from one place to another (such as a chairwhich is moved from one location to another). As discussed in greaterdetail below, the imaging system 100 is advantageously designed tooperate the pixel array 110 with adjustable resolution, frame rate,and/or bit-depth depending whether there is motion (e.g., the movingobject 50) within different portions of the field of view 30 of thepixel array 110 in such a way as to minimize a power consumption of theimaging system 100, while not losing any important visual information.

FIG. 2 shows exemplary components of one embodiment of the imagingsystem 100. Particularly, in addition to the pixel array 110, theimaging system 100 further includes a readout circuit 120, an imageprocessor 130, and a digital controller 140. As discussed in greaterdetail below, the readout circuit 120 digitizes the image framescaptured by the pixel array 110, which are processed by the imageprocessor 130 to detect objects in the image frames. The digitalcontroller 140 controls various operating parameters of the imagingsystem 100 depending on whether an object is detected in the image frameand can also synchronize the pixel array 110, the readout circuit 120,and the image processor 130.

The pixel array 110 includes a plurality of individual photodiodes orother suitable photosensitive elements, referred to herein as “pixels”112, arranged in a grid formation or other suitable configuration, eachcorresponding to a pixel of the image frames captured by the pixel array110. Each pixel 112 is configured to detect, measure, and/or record anintensity of light rays incident upon the respective pixel 112.

In the illustration of FIG. 2, only a 4×4 array of individual pixels 112is illustrated for simplicity. However, in practice, a larger array ofsome arbitrary size is generally used. FIG. 3 shows an exemplarydetailed view of the arrangement of the individual pixels 112 of thepixel array 110. Particularly, in the illustrated embodiment, aplurality of pixels 112 is arranged in a 32×32 grid formation. The arrayof individual pixels 112 is divided into a 4×4 array of “super pixels”114 (which may also be referred to herein as “tiles”), each comprising a8×8 array of individual pixels 112. It will be appreciated that that thepixel array 110 can include any arbitrary A×B array of individual pixels112. Similarly, the array of individual pixels 112 can be divided intoan arbitrary C×D array of super pixels, each comprising an arbitrary E×Farray of individual pixels 112. However, in at least on embodiment, eachsuper pixel 114 comprises a same number and arrangement of individualpixels 112.

Each super pixel 114 of the pixel array 110 is configured to operate inat least a high resolution mode and a low resolution mode. In the lowresolution mode, only a subset of the individual pixels 112 of the superpixel 114 is read out. Whereas, in the high resolution mode, all of theindividual pixels 112 of the super pixel 114 are read out. FIG. 4A showsoperation of a super pixel 114 in the low resolution mode, in which justfour of the individual pixels 112′ are read out (illustrated withdiagonal shading in the figures). In contrast, FIG. 4B shows operationof a super pixel 114 in the high resolution mode, in which all of thepixels 112′ are read out.

Returning to FIG. 2, in one embodiment, the pixel array 110 includes arow select 116 and a column readout 118 that work in concert to read outlight intensity values from individual pixels 112 in the pixel array110. Particularly, each pixel 112 is associated with one or moretransistors (not shown) configured to selectively connect the respectivepixel 112 to a corresponding vertical column bus. In one embodiment,each pixel 112 is associated with four transistors in a so-called “4Tpixel” configuration. The row select 116 is operably connected to theone or more transistors associated with each pixel 112 and is configuredto operate the transistors to selectively connect individual pixels 112and/or a particular horizontal row of pixels 112 to the correspondingvertical column buses. In at least some embodiments, the row select 116is further configured to operate the one or more transistors to resetindividual pixels 112 and/or the particular horizontal row of pixels 112before connecting them to the vertical column buses, thereby definingand/or setting the integration time (which may also be referred toherein as an “exposure time”). In at least some embodiments, the rowselect 116 is further configured to transfer collected charge from thepixels 112 to the vertical column bus during a readout operation.

The column readout 118 is configured to read out light intensity valuesfrom the individual pixels 112 and/or a particular horizontal row ofpixels 112 that are connected to the vertical column buses.Particularly, the column readout 118 is configured to provide readoutvoltages that are proportional to an amount of photons collected by eachrespective photodiode connected to a corresponding vertical column busduring the integration time and/or exposure time. The column readout 118may comprise any suitable arrangement of amplifiers, capacitors,transistors, multiplexers, etc. to provide the readout voltages. In someembodiments, the column readout 118 includes a readout amplifier circuit(not shown) connected to each vertical column bus and configured toconvert a charge build up and/or a current at the respective photodiodesinto the corresponding readout voltages. In other embodiments, the oneor more transistors associated with each pixel 112 are configured toperform the charge-to-voltage and/or current-to-voltage conversion andthe readout voltages are provided to vertical column buses directly.

The readout circuit 120 is operably connected to the column readout 118to receive the readout voltages corresponding to the light intensityvalues for the pixels 112 that are read out. The readout circuit 120includes one or more analog-to-digital converters configured to convertthe readout voltages for pixel 112 that is read out into digitized lightintensity values. It will be appreciated that, in some embodiments, theanalog-to-digital conversion can alternatively be performed byanalog-to-digital converters integrated with the column readout 118. Inthis way, the readout circuit 120 can itself be a component of the pixelarray 110 and integrated with the column readout 118 directly. Thereadout circuit 120 is configured to provide the digitized lightintensity values corresponding to each captured image frame to the imageprocessor 130.

The digital controller 140 is operably connected to the row select 116and the column readout 118 and configured to control which pixels 112 ofthe pixel array 110 are read out. The digital controller 140 has atleast one processor 142 operably connected to memory of the digitalcontroller 140 which stores program instructions 144 and is configuredto execute the program instructions 144 to at least enable said controlof which pixels 112 of the pixel array 110 are read out. In at least oneembodiment, the digital controller 140 further includes additionalmemory, such as SRAM (static random access memory) 146 for storingoperational data. It will be recognized by those of ordinary skill inthe art that a “processor” as used herein includes any hardware system,hardware mechanism or hardware component that processes data, signals,and/or other information. A processor can include a system with acentral processing unit, multiple processing units, application-specificintegrated circuits, programmable logic devices, dedicated circuitry forachieving functionality, and/or other systems. Likewise, the memoriesdiscussed herein may be of any type of device capable of storinginformation accessible by the processor, such as a memory card, ROM,RAM, hard drives, discs, flash memory, or any of various othercomputer-readable media serving as data storage devices as will berecognized by those of ordinary skill in the art.

The digital controller 140 is configured to operate each super pixel 114of the pixel array 110 in either the high resolution mode or the lowresolution mode, in response to commands from the image processor 130.Particularly, in response to a command from the image processor 130 fora particular super pixel 114 to be operated in the high resolution mode,the digital controller 140 is configured to control the row select 116and the column readout 118 to update and/or sample light intensityvalues for every pixel 112 in the particular super pixel 114. Similarly,in response to a command from the image processor 130 for a particularsuper pixel 114 to be operated in the low resolution mode, the digitalcontroller 140 is configured to control the row select 116 and thecolumn readout 118 to update and/or sample light intensity values onlyfor a defined subset of the pixels 112 in the particular super pixel114.

The particular subset of pixels 112 that are updated when a particularsuper pixel 114 is operated in the low resolution mode is defined by anadjustable read out pattern. In the example of FIG. 4A, the particularreadout pattern used in the low resolution mode includes pixels 112′ atlocations (2, 2), (2, 6), (6, 2), and (6, 6) within the super pixel 114.However, it will be appreciated that this readout pattern is merelyexemplary and that any suitable subset of the pixels 112′ may be read inthe low resolution mode. In one embodiment, the processor 142 isconfigured to store information defining the readout pattern to be usedin the low resolution mode on the SRAM 146. In at least one embodiment,the processor 142 is configured to receive commands from the imageprocessor 130 that define the readout pattern to be used. In someembodiments, the digital controller 140 is configured to operate thesuper pixels 114 of the pixel array 110 in additional resolution modes,such as an intermediate resolutions mode having a readout pattern inwhich the subset of pixels 112′ that are read out is larger than in thelow resolution mode, but does not include all of the pixels 112 as inthe high resolution mode.

Returning to FIG. 3, in the illustrated example, all super pixels 114 ofthe pixel array 110 are operated in the low resolution mode, with thesame four-pixel readout pattern of FIG. 4A. Accordingly, in thisexample, while operating in the low resolution mode, the effectiveupdate resolution of each image frame captured by the pixel array 110 is8×8, which is significantly reduced from the full 32×32 resolution ofimage frames captured by the pixel array 110 while all super pixels 114are operating in the high resolution mode. As mentioned above, each ofthe super pixels 114 can be independently operated in at least the lowresolution mode or the high resolution mode. As a result, the pixelarray 110 may be operated, in some instances, such that only certainsuper pixels 114 are operated in the high resolution mode, while othersuper pixels are operated in the low resolution mode. As a result, thenumber of pixels updated in each image frame is dynamically reduceddepending on which super pixels are operated in the low resolution modeor the high resolution mode.

In at least one embodiment, the readout circuit 120 is configured tosample the readout voltages corresponding to the light intensity valuesfor each pixel 112 at an adjustable update rate and/or frame rate and togenerate digitized light intensity values with an adjustable bit depth.In one embodiment, the readout circuit 120 is configured to generate thedigitized light intensity values and/or image frames with a particularupdate rate and/or frame rate in response to a command received from thedigital controller 140 and/or the image processor 130 indicating aparticular update rate and/or frame rate to be used. In one embodiment,the readout circuit 120 is configured to generate the digitized lightintensity values and/or image frames with a particular bit depth inresponse to a command received from the digital controller 140 and/orthe image processor 130 indicating a particular bit depth to be used.

In at least one embodiment, the digital controller 140 receives acommand from the image processor 130 indicating a reduced frame rateand/or bit depth to be for super pixels 114 operated in the lowresolution mode. In one embodiment, information regarding the reducedframe rate and/or bit depth to be used in the low resolution mode isstored on the SRAM 146. If a particular super pixel 114 is to beoperated in the high resolution mode, the digital controller 140commands the readout circuit 120 to generate digitized light intensityvalues corresponding to pixels 112 of the particular super pixel 114with a maximum frame rate and a maximum bit depth. In contrast, if aparticular super pixel 114 is to be operated in the low resolution mode,the digital controller 140 commands the readout circuit 120 to generatedigitized light intensity values corresponding to pixels 112 of theparticular super pixel 114 with the reduced frame rate and/or thereduced bit depth. In this way, the amount of analog-to-digitalconversion that must be performed by the readout circuit 120 isdynamically reduced depending on which super pixels 114 are operated inthe low resolution mode or the high resolution mode.

The image processor 130 is configured to receive digitized lightintensity values and/or image frames from the readout circuit 120. Theimage processor 130 has at least one processor 132 operably connected tomemory of the image processor 130 which stores program instructions 134and is configured to execute the program instructions 134 to enable oneor more features of the image processor 130, at least includingdetecting moving objects in the received image frames. In at least oneembodiment, the image processor 130 further includes additional memory,such as SRAM (static random access memory) 136 for storing operationaldata. It will be recognized by those of ordinary skill in the art that a“processor” as used herein includes any hardware system, hardwaremechanism or hardware component that processes data, signals, and/orother information. A processor can include a system with a centralprocessing unit, multiple processing units, application-specificintegrated circuits, programmable logic devices, dedicated circuitry forachieving functionality, and/or other systems. Likewise, the memoriesdiscussed herein may be of any type of device capable of storinginformation accessible by the processor, such as a memory card, ROM,RAM, hard drives, discs, flash memory, or any of various othercomputer-readable media serving as data storage devices as will berecognized by those of ordinary skill in the art.

The program instructions 134 include instructions corresponding to abackground removal module 138. Particularly, the processor 132 isconfigured to receive a plurality of image frames and/or digitized lightintensity values thereof from the readout circuit 120. For each imageframe, the processor 132 is configured to detect moving objects (e.g.,the moving object 50 of FIG. 1) present in the respective image frame.Particularly, in at least one embodiment, the processor 132 isconfigured to detect whether a moving object is present in eachindividual super pixel 114 of the respective image frame. The processor132 is configured to store, in the SRAM 136, information regarding thedetected objects in the image frame and/or in each super pixel 114.

The processor 132 is configured to execute program instructions of thebackground removal module 138 to determine whether each super pixel 114in the pixel array 110 should be operated in the low resolution mode orthe high resolution mode. The processor 132 is configured to inform thedigital controller 140 of which super pixels 114 are to be operated inthe low resolution mode or the high resolution mode. In response todetermining that a moving object is present in a particular super pixel114, the processor 132 is configured to determine that the respectivesuper pixel 114 is to be operated in the high resolution mode.Otherwise, in response to determining that no moving objects are presentin a particular super pixel 114, the processor 132 is configured todetermine that the respective super pixel 114 is to be operated in thelow resolution mode. In this way, super pixels 114 having moving objectsdetected therein are operated in the high resolution mode, whereas eachother super pixel is operated in the low resolution mode.

In at least one embodiment, in order to detect moving objects in acurrent image frame, the processor 132 is configured to compare thecurrent image frame with a reference background image frame and/orbackground model that is stored in the SRAM 136. In some embodiments,the processor 132 is configured to calculate the intermediate imageframe having a plurality of light intensity difference values bysubtracting a reference background image frame from the current imageframe. As a result, pixels of the intermediate image frame correspondingto pixels of the current image frame having light intensity values thatare the same or essentially similar to that of the background imageframe have very small values or zero values. However, pixels of theintermediate image frame corresponding to pixels of the current imageframe having light intensity values that are different than that of thebackground image frame have larger non-zero values. The portions of theintermediate image frame having the larger non-zero values correspond toobjects that are present in the current image frame that are not presentin the background image frame. Thus, these portions of the intermediateimage frame can reasonably be assumed to include a moving object, suchas the moving object 50 of FIG. 1.

For each super pixel 114, if the corresponding pixels of theintermediate image frame include the larger non-zero valuescorresponding to moving objects, then the processor 132 is configured todetermine that the respective super pixel 114 is to be operated in thehigh resolution mode. Otherwise, if the corresponding pixels of theintermediate frame include only the very small values or zero valuescorresponding to the background environment, then the processor 132 isconfigured to determine that the respective super pixel 114 is to beoperated in the low resolution mode. In at least one embodiment, foreach super pixel 114, the processor 132 is configured to determine thatthe respective super pixel 114 is to be operated in the high resolutionmode in response to any corresponding pixel in the intermediate framehaving an absolute value that exceeds a predetermined thresholddifference value. Otherwise, the processor 132 is configured todetermine that the respective super pixel 114 is to be operated in thelow resolution mode in response to none of the corresponding pixels inthe intermediate frame having an absolute value that exceeds thepredetermined threshold difference value.

In at least one embodiment, the program instructions 134 includeinstructions corresponding to a learning module 139. Particularly, theprocessor 132 is configured to learn the background image frame andupdate it over time, as needed. In one embodiment, the background imageframe may be learned initially during a setup phase and updated asneeded. In one embodiment, the processor 132 is configured to update thebackground image frame on a super pixel basis. In one embodiment, inresponse to pixels of the intermediate image frame corresponding to arespective super pixel having consistent and stable non-zero values fora predetermined number of image frames and/or for a predeterminedduration of time, the processor 132 is configured to update that superpixel of the background image frame with the current values of thecurrent image frame. This may occur, for example, if one of thebackground objects 40 (e.g., a chair) is moved to a slightly differentlocation of the scene 20 and left at the new location semi-permanently.When a particular super pixel 114 is updated, the super pixel 114 isoperated for at least one frame in the high resolution mode and thebackground image frame is updated in high resolution. After updating thebackground image frame, the processor 132 is configured to store theupdated background image frame in the SRAM 136. In some embodiments, theprocessor 132 executes instructions of the learning module 139 todetermine an identity or type of the moving objects and storeinformation regarding the identity or type in the SRAM 136.

In some embodiments, the processor 132 is configured to executeinstructions of the background removal module 138 and/or learning module139 to minimize a need to update the background image frame and/orbackground model due to changes in lighting conditions. Particularly,significant changes in lighting conditions in the scene 20 willgenerally trigger an updating of the background image frame due tosignificant changes in the light intensity values in the received imageframes. The digital controller 140 is configured to command the pixelarray 110 to operate with an adjustable integration time or exposuretime. In at least one embodiment, the processor 132 is configuredcommand the digital controller 140 to adjust the integration time orexposure time in response to detecting a lighting change in the currentimage frame. Particularly, if the scene 20 becomes brighter, then theprocessor 132 is configured to command the digital controller 140 toreduce the integration time or exposure time. In contrast, if the scene20 becomes darker, then the processor 132 is configured to command thedigital controller 140 to increase the integration time or exposuretime. It at least one embodiment, the processor 132 is configured todetect a lighting change in response to all or a predeterminedpercentage of the pixels of the current image frame being different fromthe background image frame by roughly a same percentage. In this way,the changes in lighting conditions can be compensated for withoutneeding to update the background image frame. The adjustment ofintegration time or exposure time will generally result in less frequentupdates to the background image frame and additional power savings.

The above description describes various features of the image processor130 and digital controller 140. It will be appreciated that, in someembodiments, the functions described with respect to the processor 132and 142 of the image processor 130 and digital controller 140 and/or theprogram instructions 134 and 144 of the image processor 130 and digitalcontroller 140 may be performed by a single processing device.Similarly, in some embodiments, features described with respect to theprocessor 132 and/or the program instructions 134 of the image processor130 may be performed by the processor 142 and/or the programinstructions 144 of the digital controller 140. Likewise, in someembodiments, features described with respect to the processor 142 and/orthe program instructions 144 of the digital controller 140 may beperformed by the processor 132 and/or the program instructions 134 ofthe image processor 130.

The digital controller 140 and image processor 130, and the programinstructions associated therewith, the improve upon conventionalcontrollers for pixels arrays because they enable dynamic adjustment ofthe resolution, the update/refresh rate, and bit depth of individualeach super pixel 114 depending on a moving object is present in therespective super pixels 114. By doing this, much less data is read out,converted, and processed, thereby reducing the power consumption of theimaging 100. However, because there are still selected pixels read outin the low resolution mode, the processor 132 is still able to detect ifa moving object appears in the super pixels 114 operated in the lowresolution mode. Thus, energy savings are achieved without the loss ofimportant information.

While the disclosure has been illustrated and described in detail in thedrawings and foregoing description, the same should be considered asillustrative and not restrictive in character. It is understood thatonly the preferred embodiments have been presented and that all changes,modifications and further applications that come within the spirit ofthe disclosure are desired to be protected.

Embodiments within the scope of the disclosure may also includenon-transitory computer-readable storage media or machine-readablemedium for carrying or having computer-executable instructions or datastructures stored thereon. Such non-transitory computer-readable storagemedia or machine-readable medium may be any available media that can beaccessed by a general purpose or special purpose computer. By way ofexample, and not limitation, such non-transitory computer-readablestorage media or machine-readable medium can comprise RAM, ROM, EEPROM,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium which can be used to carryor store desired program code means in the form of computer-executableinstructions or data structures. Combinations of the above should alsobe included within the scope of the non-transitory computer-readablestorage media or machine-readable medium.

Embodiments may also be practiced in distributed computing environmentswhere tasks are performed by local and remote processing devices thatare linked (either by hardwired links, wireless links, or by acombination thereof) through a communications network.

Computer-executable program instructions include, for example,instructions and data which cause a general purpose computer, specialpurpose computer, or special purpose processing device to perform acertain function or group of functions. Computer-executable instructionsalso include program modules that are executed by computers instand-alone or network environments. Generally, program modules includeroutines, programs, objects, components, and data structures, etc. thatperform particular tasks or implement particular abstract data types.Computer-executable instructions, associated data structures, andprogram modules represent examples of the program code means forexecuting steps of the methods and features disclosed herein. Theparticular sequence of such executable instructions or associated datastructures represents examples of corresponding acts for implementingthe functions described in such steps.

What is claimed is:
 1. An imaging system comprising: a pixel arrayhaving a plurality of photosensitive elements arranged in a gridformation and being divided into a plurality of groups of photosensitiveelements, the photosensitive elements in each group of photosensitiveelements being adjacent to one another in the grid formation, the pixelarray being configured to output readout voltages proportional to lightintensities at individual photosensitive elements in the plurality ofphotosensitive elements; an analog-to-digital converter operablyconnected to the pixel array to receive the readout voltages and convertthe readout voltages into digitized light intensity values; and at leastone processor operably connected to the pixel array and theanalog-to-digital converter, the at least one processor being configuredto selectively operate each group of photosensitive elements in theplurality of groups of photosensitive elements in one of (i) a highresolution mode in which the pixel array outputs readout voltagescorresponding to all of the photosensitive elements in the respectivegroup of photosensitive elements and (ii) a low resolution mode in whichthe pixel array outputs readout voltages corresponding to only a subsetof the photosensitive elements in the respective group of photosensitiveelements, wherein each group of photosensitive elements in the pluralityof groups of photosensitive elements is operable by the at least oneprocessor in both the high resolution mode and the low resolution modeat different times.
 2. The imaging system of claim 1, the at least oneprocessor being configured to: receive an image frame, from theanalog-to-digital converter, comprising digitized light intensity valuescorresponding to at least some of the photosensitive elements in eachgroup of photosensitive elements in the plurality of groups ofphotosensitive elements; and selectively operate each group ofphotosensitive elements in the plurality of groups of photosensitiveelements in the high resolution mode and the low resolution mode atdifferent times depending on the digitized light intensity values of theimage frame.
 3. The imaging system of claim 2, the at least oneprocessor being configured to: operate a first group of photosensitiveelements in the plurality of groups of photosensitive elements in thelow resolution mode depending on to the digitized light intensity valuesin the image frame corresponding to the photosensitive elements in thefirst group of photosensitive elements; and operate a second group ofphotosensitive elements in the plurality of groups of photosensitiveelements in the high resolution mode depending on to the digitized lightintensity values in the image frame corresponding to the photosensitiveelements in the second group of photosensitive elements.
 4. The imagingsystem of claim 2, the at least one processor being configured to, foreach group of photosensitive elements in the plurality of groups ofphotosensitive elements: detect whether any moving objects are presentin a respective portion of the image frame corresponding to therespective group of photosensitive elements based on the digitized lightintensity values in the image frame corresponding to the photosensitiveelements in the respective group of photosensitive elements; operate therespective group of photosensitive elements in the low resolution modein response to no moving object being detected in the respective portionof the image frame corresponding to the respective group ofphotosensitive elements; and operate the respective group ofphotosensitive elements in the high resolution mode in response to amoving object being detected in the respective portion of the imageframe corresponding to the respective group of photosensitive elements.5. The imaging system of claim 4, the at least one processor beingconfigured to, for each group of photosensitive elements in theplurality of groups of photosensitive elements: compare the respectiveportion of the image frame corresponding to the respective group ofphotosensitive elements with a corresponding portion of a referencebackground image frame to detect whether any moving objects are presentin the respective portion of the image frame.
 6. The imaging system ofclaim 5, the at least one processor being configured to: generate anintermediate image frame comprising light intensity difference values bysubtracting the reference background image frame from the received imageframe.
 7. The imaging system of claim 6, the at least one processorbeing configured to, for each group of photosensitive elements in theplurality of groups of photosensitive elements: operate the respectivegroup of photosensitive elements in the low resolution mode in responseto the absolute value of each light intensity difference value of arespective portion of the intermediate image frame that corresponds tothe respective group of photosensitive elements being less than apredetermined threshold difference value; and operate the respectivegroup of photosensitive elements in the high resolution mode in responseto the absolute value of any light intensity difference value of arespective portion of the intermediate image frame that corresponds tothe respective group of photosensitive elements being greater than apredetermined threshold difference value.
 8. The imaging system of claim5, the at least one processor being configured to: detect a change oflighting conditions in the image frame based on the digitized lightintensity values of the image frame; and operate the pixel array with anadjusted exposure time that depends on the change in the lightingconditions.
 9. The imaging system of claim 6, the at least one processorbeing configured to: identify at least one portion of the referencebackground image frame to be updated based on the intermediate imageframe, the at least one portion corresponding to at least one group ofphotosensitive elements in the plurality of groups of photosensitiveelements; and update the at least one portion of the referencebackground image frame based on at least one corresponding portion ofthe image frame.
 10. The imaging system of claim 1, wherein: each groupof photosensitive elements in the plurality of groups of photosensitiveelements comprises a common number of photosensitive elements arrangedin a common formation within the grid formation; and for each group ofphotosensitive elements in the plurality of groups of photosensitiveelements that is operated in the low resolution mode, the subset of thephotosensitive elements for which the pixel array outputs readoutvoltages corresponds to a defined pattern of photosensitive elementswithin the common formation.
 11. The imaging system of claim 1, whereinthe defined pattern of photosensitive elements within the commonformation is adjustable by the at least one processor.
 12. The imagesystem of claim 1, the at least one processor being configured to: foreach group of photosensitive elements in the plurality of groups ofphotosensitive elements that is operated in the low resolution mode,control the analog-to-digital converter to convert the readout voltagescorresponding to the respective the subset of the photosensitiveelements in the respective group of photosensitive elements with a firstbit depth that is less than a maximum bit depth of the analog-to-digitalconverter; and for each group of photosensitive elements in theplurality of groups of photosensitive elements that is operated in thehigh resolution mode, control the analog-to-digital converter to convertthe readout voltages corresponding to all of the photosensitive elementsin the respective group of photosensitive elements with the maximum bitdepth of the analog-to-digital converter.
 13. The imaging system ofclaim 12, wherein the first bit depth is adjustable by the at least oneprocessor.
 14. The image system of claim 1, the at least one processorbeing configured to: for each group of photosensitive elements in theplurality of groups of photosensitive elements that is operated in thelow resolution mode, control the pixel array and the analog-to-digitalconverter to update and convert the readout voltages corresponding tothe subset of the photosensitive elements in the respective group ofphotosensitive elements with a first update rate that is less than amaximum update rate; and for each group of photosensitive elements inthe plurality of groups of photosensitive elements that is operated inthe high resolution mode, control the pixel array and theanalog-to-digital converter to update and convert the readout voltagescorresponding to all of the photosensitive elements in the respectivegroup of photosensitive elements with the maximum update rate.
 15. Theimaging system of claim 14, wherein the first update rate is adjustableby the at least one processor.